Method that expedites playing sound of a talking emoji

ABSTRACT

A method expedites playing sound of a talking emoji from a first person with a first portable electronic device (PED) to a second person with a second PED. The second PED receives the talking emoji in mono sound and convolves the mono sound into binaural sound before receiving a request to play the sound to the second user. The second PED then plays the sound of the talking emoji in binaural sound after receiving the request from the second user.

BACKGROUND

Three-dimensional (3D) sound localization offers people a wealth of newtechnological avenues to not merely communicate with each other but alsoto communicate with electronic devices, software programs, andprocesses.

As this technology develops, challenges will arise with regard to howsound localization integrates into the modern era. Example embodimentsoffer solutions to some of these challenges and assist in providingtechnological advancements in methods and apparatus using 3D soundlocalization.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a method that expedites playing sound of a graphicalrepresentation sent from a first person with a first portable electronicdevice (PED) to a second person with a second PED in accordance with anexample embodiment.

FIG. 2 is a method that expedites playing of sound to a user byprefetching, decrypting, and/or caching the sound before the sound isplayed to the listener in accordance with an example embodiment.

FIG. 3 is a method that expedites playing of sound to a user by storingmultiple versions of the sound in memory in accordance with an exampleembodiment.

FIG. 4A is a graphical representation in accordance with an exampleembodiment.

FIG. 4B is another graphical representation in accordance with anexample embodiment.

FIG. 4C is another graphical representation in accordance with anexample embodiment.

FIG. 4D is another graphical representation in accordance with anexample embodiment.

FIG. 4E is another graphical representation in accordance with anexample embodiment.

FIG. 4F is another graphical representation in accordance with anexample embodiment.

FIG. 5 is an example computer system in accordance with an exampleembodiment.

FIG. 6 is an example of an electronic device in accordance with anexample embodiment.

SUMMARY

Example embodiments include methods and apparatus that expediteprocessing and/or playing of binaural sound to a listener.

During an electronic communication between a first user and a seconduser, an electronic device processes or convolves sound into binauralsound for the second user before the second user requests the sound tobe heard. In this way, processing or playing of binaural sound isexpedited since the binaural sound is already convolved and ready toplay when the second user is ready to hear the sound.

Other example embodiments are discussed herein.

DETAILED DESCRIPTION

Binaural sound or three-dimensional (3D) sound externally localizes awayfrom a head of the listener, unlike stereo or mono sound that localizesinside the head of the listener or localizes to a physical soundspeaker. Thus, when a listener hears binaural sound, a source orlocation of the sound occurs outside the head of the listener eventhough this location may be in empty space or space not occupied with aphysical sound speaker or loud speaker.

Binaural sound has many technical challenges and problems, especiallywhen users exchange or play binaural sound during an electroniccommunication. Example embodiments offer solutions to these challengesand problems.

One problem during an electronic communication is that processing orconvolving sound with head-related transfer functions (HRTFs) is processintensive. Electronic devices often use a dedicated or specializedprocessor, such as a digital signal processor (DSP), to perform the taskof convolving sound into binaural sound. This task can be especiallyprocess intensive if the head of the listener is moving, the sound ismoving with respect to the listener, or the sound must be repeatedlyconvolved with different pairs of HRTFs. If the processor in theelectronic device cannot convolve the sound quickly enough, the soundthe listener hears may appear delayed or jumpy and ultimately diminishthe experience of hearing binaural sound.

Further yet, convolving sound with HRTFs is time-consuming. If a useractivates playing of 3D sound, the user must wait while the processorconvolves the sound into binaural sound. This process can delay playingthe sound to the user.

Example embodiments solve these problems and others.

In one example embodiment, a method expedites playing sound of agraphical representation sent or provided from a first person with afirst portable electronic device (PED) to a second person with a secondPED. The graphical representation includes sound (such as a sound clip,sound file, or audio file) that plays to the second person. The secondPED receives the graphical representation and the sound in mono sound orstereo sound and convolves this sound into binaural sound beforereceiving a request to play the sound to the second person. Thisconvolution occurs in anticipation of the sound being played to thesecond person at a future point in time. When the second personsubsequently activates the graphical representation or otherwise playsthe sound received from the first person, the sound immediately orinstantly plays as binaural sound since the sound was already convolvedinto the binaural sound before the request or before activation of thegraphical representation. As such, the second person does not have towait while a processor (such as a DSP) convolves the sound from mono orstereo sound into binaural sound since the sound was already convolvedand stored on the electronic device of the second person. Processing orconvolving the sound before the sound is requested expedites playing ofthe sound to the second person and enhances the user experiencelistening to binaural sound.

Consider another example in which the electronic device of the persondownloads or receives sound in mono sound or stereo sound. This sound isnot yet convolved into binaural sound. With a conventional technique,the electronic device would not convolve the sound until it is actuallyrequested (e.g., at a point in time when a user requests to hear thesound or a software program elects to play the sound to the user).Convolving the sound at this time, however, is process intensive andtime-consuming. Instead, an example embodiment convolves the soundbefore the sound is actually needed for playback or before the user orsoftware program requests the sound. By convolving the sound earlierbefore it is needed or requested, the sound plays immediately uponrequest. No time delay occurs since the convolution is already complete,and the sound is available and ready for immediate playing to thelistener.

An example embodiment predicts or anticipates that the listener willrequest to hear the sound as binaural sound at a point in time in thefuture. By convolving the sound before this point in time, exampleembodiments expedite playing and/or processing of the sound when thesound is actually requested for play. In this way, the binaural sound isalready convolved and ready to play upon activation (e.g., when a useractivates a graphical representation that includes sound or activates asound file or sound clip). This process expedites playing of thebinaural sound at the electronic device of the user since thiselectronic device is not required to convolve the sound into binauralsound at the point in time when the user hears the sound. The binauralsound stored in the electronic device is ready for immediate play at theelectronic device of the user.

Example embodiments also expedite playing of binaural sound byprefetching, decrypting, caching, and/or storing multiple versions ofthe sound as discussed herein.

FIG. 1 is a method that expedites playing sound of a graphicalrepresentation sent from a first person with a first portable electronicdevice (PED) to a second person with a second PED in accordance with anexample embodiment.

Block 100 states transmit, from a first electronic device or a server incommunication with the first electronic device, a graphicalrepresentation with mono sound or stereo sound to a second electronicdevice.

The first electronic device or the server transmits the graphicalrepresentation and sound over one or more wired or wireless networks(e.g., a cellular network, the internet, etc.) to the second electronicdevice. For example, the first electronic device or server includes awireless transmitter/receiver that sends the graphical representationand sound.

Consider an example in which a first user commands or instructs a soundclip to play to a second user during an electronic communication betweenthe first and second users. For example, the first and second users aretalking in a telephone call and/or exchanging text messages. In responseto this command or instruction, the first electronic device transmitsthe sound clip and a graphical representation to the second electronicdevice.

In another example embodiment, a server or another electronic devicetransmits the sound to the second electronic device. Consider an examplein which the first and second users talk or message each other with amobile messaging software application. The application executes on theelectronic devices and one or more servers. When the first user clickson a 3D sound emoji, this action causes one of the servers or electronicdevices to transmit the 3D emoji and sound to the second electronicdevice.

Block 110 states receive, at the second electronic device, the graphicalrepresentation and the sound in mono sound or stereo sound.

The second electronic device receives the graphical representation andsound from the first electronic device or another electronic device(e.g., a server) in communication with the first electronic. Forexample, the second electronic device includes a wirelesstransmitter/receiver that receives the sound over one or more networks.

Block 120 states convolve and/or process, with a processor, the monosound or stereo sound with head-related transfer functions (HRTFs) tochange the mono sound or stereo sound into binaural sound before thesound of the graphical representation plays to the second person.

The processor, processors, or processing unit processes or convolves thesound before the second person, an electronic device, or a softwareprogram activates the sound to play to the second person. Convolutionoccurs before the sound plays to the second person so the second personhears the sound as binaural sound and not as mono sound or stereo soundas received from the first electronic device.

For example, a processor (such as a DSP) processes or convolves thesound with one or more of head-related transfer functions (HRTFs),head-related impulse responses (HRIRs), room impulse responses (RIRs),room transfer functions (RTFs), binaural room impulse responses (BRIRs),binaural room transfer functions (BRTFS), interaural time delays (ITDs),interaural level differences (ITDs), and a sound impulse response.

An example embodiment processes or convolves the sound with the HRTFsafter the sound is provided to the electronic device of the second userbut before the second user hears the sound or requests to hear thesound. This expedites processing and/or playing of the sound to thesecond user since the second user does not have to wait while aprocessor processes or convolves the sound into binaural sound when thesecond user subsequently makes a request to hear the sound.

Sound includes, but is not limited to, one or more of stereo sound, monosound, binaural sound, computer-generated sound, sound captured withmicrophones, and other sound. Furthermore, sound includes differenttypes including, but not limited to, music, background sound orbackground noise, human voice, computer-generated voice, and othernaturally occurring or computer-generated sound.

When the sound is recorded or generated in mono sound or stereo sound,convolution changes the sound to binaural sound. For example, one ormore microphones record a human person speaking in mono sound or stereosound, and a processor processes this sound with filters to change thesound into binaural sound.

The processor or sound hardware processing or convolving the sound canbe located in one or more electronic devices or computers including, butnot limited to, headphones, smartphones, tablet computers, electronicspeakers, head mounted displays (HMDs), optical head mounted displays(OHMDs), electronic glasses (e.g., glasses that provide augmentedreality (AR)), servers, portable electronic devices (PEDs), handheldportable electronic devices (HPEDs), wearable electronic devices (WEDs),and other portable and non-portable electronic devices. These electronicdevices can also be used to execute example embodiments.

In one example embodiment, the DSP is located in the electronic deviceof the second user. In other example embodiments, the DSP is located inother electronic devices, such as a server in communication with thefirst and second electronic devices.

The DSP processes or convolves stereo sound or mono sound with a processknown as binaural synthesis or binaural processing to provide the soundwith sound localization cues (ILD, ITD, and/or HRTFs) so the listenerexternally localizes the sound as binaural sound or 3D sound.

An example embodiment models the HRTFs with one or more filters, such asa digital filter, a finite impulse response (FIR) filter, an infiniteimpulse response (IIR) filter, etc. Further, an ITD can be modeled as aseparate delay line.

When the binaural sound is not captured (e.g., on a dummy head or humanhead), the captured sound is convolved with sound localizationinformation (SLI). This information includes one or more of HRTFs,HRIRs, BRTFs, BRIRs, ILDs, ITDs, and/or other information discussedherein. By way of example, SLI are retrieved, obtained, or received frommemory, a database, a file, an electronic device (such as a server,cloud-based storage, or another electronic device in the computer systemor in communication with a PED providing the sound to the user throughone or more networks), etc. Instead of being retrieved from memory, thisinformation can also be calculated in real-time.

A central processing unit (CPU), processor (such as a DSP), ormicroprocessor processes and/or convolves the sound with the SLI, suchas a pair of head related transfer functions (HRTFs), ITDs, and/or ILDsso that the sound will localize to a zone, area, or sound localizationpoint (SLP). For example, the sound localizes to a specific point (e.g.,localizing to point (r, θ, ϕ)) or a general location or area (e.g.,localizing to far-field location (θ, ϕ) or near-field location (θ, ϕ)).As an example, a lookup table that stores a set of HRTF pairs includes afield/column that specifies the coordinates associated with each pair,and the coordinates indicate the location for the origination of thesound. These coordinates include a distance (r) or near-field orfar-field designation, an azimuth angle (θ), and/or an elevation angle(ϕ).

The complex and unique shape of the human pinnae transforms sound wavesthrough spectral modifications as the sound waves enter the ear. Thesespectral modifications are a function of the position of the source ofsound with respect to the ears along with the physical shape of thepinnae that together cause a unique set of modifications to the soundcalled head related transfer functions or HRTFs. A unique pair of HRTFs(one for the left ear and one for the right ear) can be modeled ormeasured for each position of the source of sound with respect to alistener as the customized HRTFs.

A HRTF is a function of frequency (f) and three spatial variables, byway of example (r, θ, ϕ) in a spherical coordinate system. Here, r isthe radial distance from a recording point where the sound is recordedor a distance from a listening point where the sound is heard to anorigination or generation point of the sound; θ (theta) is the azimuthangle between a forward-facing user at the recording or listening pointand the direction of the origination or generation point of the soundrelative to the user; and ϕ (phi) is the polar angle, elevation, orelevation angle between a forward-facing user at the recording orlistening point and the direction of the origination or generation pointof the sound relative to the user. By way of example, the value of (r)can be a distance (such as a numeric value) from an origin of sound to arecording point (e.g., when the sound is recorded with microphones) or adistance from a SLP to a head of a listener (e.g., when the sound isgenerated with a computer program or otherwise provided to a listener).

When the distance (r) is greater than or equal to about one meter (1 m)as measured from the capture point (e.g., the head of the person) to theorigination point of a sound, the sound attenuates inversely with thedistance. One meter or thereabout defines a practical boundary betweennear-field and far-field distances and corresponding HRTFs. A“near-field” distance is one measured at about one meter or less;whereas a “far-field” distance is one measured at about one meter ormore. Example embodiments are implemented with near-field and far-fielddistances.

The coordinates for external sound localization can be calculated orestimated from an interaural time difference (ITD) of the sound betweentwo ears. ITD is related to the azimuth angle according to, for example,the Woodworth model that provides a frequency independent ray tracingmethodology. The coordinates (r, θ, ϕ) for external sound localizationcan also be calculated from a measurement of an orientation of and adistance to the face of the person when a head related impulse response(HRIR) is captured.

The coordinates can also be calculated or extracted from one or moreHRTF data files, for example by parsing known HRTF file formats, and/orHRTF file information. For example, HRTF data is stored as a set ofangles that are provided in a file or header of a file (or in anotherpredetermined or known location of a file or computer readable medium).The data can include one or more of time domain impulse responses (FIRfilter coefficients), filter feedback coefficients, and an ITD value.This information can also be referred to as “a” and “b” coefficients. Byway of example, these coefficients are stored or ordered according tolowest azimuth to highest azimuth for different elevation angles. TheHRTF file can also include other information, such as the sampling rate,the number of elevation angles, the number of HRTFs stored, ITDs, a listof the elevation and azimuth angles, a unique identification for theHRTF pair, and other information. The data can be arranged according toone or more standard or proprietary file formats, such as AES69, andextracted from the file.

The coordinates and other HRTF information can be calculated orextracted from the HRTF data files. A unique set of HRTF information(including r, θ, ϕ) is determined for each unique HRTF.

The coordinates and other HRTF information are also stored in andretrieved from memory, such as storing the information in a look-uptable. The information is quickly retrieved to enable real-timeprocessing and convolving of sound using HRTFs and hence improvescomputer performance of execution of binaural sound.

The SLP represents a location where a person will perceive an origin ofthe sound. For an external localization, the SLP is away from the person(e.g., the SLP is away from but proximate to the person or away from butnot proximate to the person). The SLP can also be located inside thehead of the person (e.g., when the sound is provided as mono sound orstereo sound). Sound can also switch between externally localizing andinternally localizing, such as appearing to move and pass through a headof a listener.

SLI can also be approximated or interpolated based on known data orknown SLI, such as SLI for other coordinate locations. For example, aSLP is desired to localize at coordinate location (2.0 m, 0°, 40°), butHRTFs for the location are not known. HRTFs are known for twoneighboring locations, such as known for (2.0 m, 0°, 35°) and (2.0 m,0°, 45°), and the HRTFs for the desired location of (2.0 m, 0°, 40°) areapproximated from the two known locations. These approximated HRTFs areprovided to convolve sound to localize at the desired coordinatelocation (2.0 m, 0°, 40°).

Sound is convolved either directly in the time domain with a finiteimpulse response (FIR) filter or with a Fast Fourier Transform (FFT).For example, an electronic device convolves the sound to one or moreSLPs using a set of HRTFs, HRIRs, BRIRs, or RIRs and provides the personwith binaural sound.

In an example embodiment, convolution involves an audio input signal andone or more impulse responses of a sound originating from variouspositions with respect to the listener. The input signal is a limitedlength audio signal (such as a pre-recorded digital audio file or soundclip) or an ongoing audio signal (such as sound from a microphone orstreaming audio over the Internet from a continuous source). The impulseresponses are a set of HRIRs, BRIRs, RIRs, etc.

Convolution applies one or more FIR filters to the input signals andconvolves the input signals into binaural audio output or binauralstereo tracks. For example, the input signals are convolved intobinaural audio output that is specific or individualized for thelistener based on one or more of the impulse responses to the listener.

The FIR filters are derived binaural impulse responses. Alternatively oradditionally, the FIR filters are obtained from another source, such asgenerated from a computer simulation or estimation, generated from adummy head, retrieved from storage, computed based on known impulseresponses captured from people, etc. Further, convolution of an inputsignal into binaural output can include sound with one or more ofreverberation, single echoes, frequency coloring, and spatialimpression.

Processing of the sound also includes calculating and/or adjusting aninteraural time difference (ITD), an interaural level difference (ILD),and/or other aspects of the sound in order to alter the cues andartificially alter the point of localization. Consider an example inwhich the ITD is calculated for a location (θ, ϕ) with discrete Fouriertransforms (DFTs) calculated for the left and right ears. The ITD islocated at the point for which the function attains its maximum value,known as the argument of the maximum or arg max as follows:

${ITD} = {\arg\mspace{14mu}{\max(\tau)}{\sum\limits_{n}{{d_{l,\theta,\phi}(n)} \cdot {{d_{r,\theta,\phi}( {n + \tau} )}.}}}}$

Subsequent sounds are filtered with the left HRTF, right HRTF, and/orITD so that the sound localizes at (r, θ, ϕ). Such sounds includefiltering stereo and monaural sound to localize at (r, θ, ϕ). Forexample, given an input signal as a monaural sound signal s(n), thissound is convolved to appear at (θ, ϕ) when the left ear is presentedwith:s _(l)(n)=s(n−ITD)·d _(l,θ,ϕ)(n);

and the right ear is presented with:s _(r)(n)=S(n)·d _(r,θϕ)(n).

Consider an example in which a dedicated digital signal processor (DSP)executes frequency domain processing to generate real-time convolutionof monophonic sound to binaural sound.

By way of example, a continuous audio input signal x(t) is convolvedwith a linear filter of an impulse response h(t) to generate an outputsignal y(t) as follows:

${y(\tau)} = {{{x(\tau)} \cdot {h(\tau)}} = {\int\limits_{0}^{\infty}{{x( {\tau - t} )} \cdot {h(t)} \cdot {{dt}.}}}}$

This reduces to a summation when the impulse response has a given lengthN and the input signal and the impulse response are sampled at t=iDt asfollows:

${y(i)} = {\sum\limits_{j = 0}^{N - 1}\;{{x( {i - j} )} \cdot {{h(j)}.}}}$

Execution time of convolution further reduces with a Fast FourierTransform (FFT) algorithm and/or Inverse Fast Fourier Transform (IFFT)algorithm.

Consider another example of binaural synthesis in which recorded orsynthesized sound is filtered with a binaural impulse response (e.g.,HRIR or BRIR) to generate a binaural output sound to the person. Theinput sound is preprocessed to generate left and right audio streamsthat are mapped to one or more sound sources or sound localizationpoints (known as SLPs). These streams are convolved with a binauralimpulse response for the left ear and the right ear to generate the leftand right binaural output sound signal. The output sound signal isfurther processed depending on a final destination. For example, across-talk cancellation algorithm is applied to the output sound signalwhen it will be provided through loudspeakers or applying artificialbinaural reverberation to provide 3D spatial context to the sound.

Block 130 states receive, at the second electronic device and after theprocessor already convolved the mono sound or stereo sound into thebinaural sound, a request or activation of the sound.

The second user, an electronic device, or a software program activatesthe graphical representation and/or plays the binaural sound to thesecond user. For example, the second user requests or activates thegraphical representation and/or sound after the sound was alreadyconvolved. For instance, the second user clicks on the graphicalrepresentation, issues a voice command to play the sound or activate thegraphical representation, uses a mouse or pointer to activate or playthe sound, commands or instructs a software program to activate or playthe sound, issues body gesture (e.g., hand gesture, eye movement, etc.),etc. Activation or playing of the sound can occur in other ways as well.For example, the sound plays when the second person views the graphicalrepresentation, opens or enlarges a window, or opens a software program.For example, the sound plays upon occurrence of another event, such asplaying at a certain time of day, playing when the user proceeds to ageographical or internet of things (IoT) location, the user enters avirtual space, the user focuses a window, the user dons a PED, the useractivates a program, the user turns on or awakes from sleep anelectronic device, or other events discussed herein.

Block 140 states play, to the second user with the second electronicdevice and in response to the request or activation, the sound processedand/or convolved with the HRTFs such that the sound externally localizesas binaural sound away from a head of the second user.

The sound plays to the listener as binaural sound that externallylocalizes away from or outside of the head of the listener. For example,headphones or earphones provide this sound at one or more soundlocalization points (SLPs).

Consider an example in which a first user sends a graphicalrepresentation and mono sound to an electronic device of a second user.Before the sound arrives at the electronic device of the second user,the sound transmits to a server. The server convolves the mono soundinto binaural sound and transmits this convolved sound to the electronicdevice of the second user.

Consider an example in which the first user desires to send a graphicalrepresentation and sound to the electronic device of the second user.The graphical representation and/or sound are also stored on a serverwhich also stores or has access to the HRTFs of the second user. Theserver convolves the sound into binaural sound and transmits thisconvolved sound to the electronic device of the second user.

The electronic device of the first user can transmit the graphicalrepresentation and/or sound to the server that convolves the sound.Alternatively, the electronic device of the first user transmits arequest to send the graphical representation and/or sound to the secondelectronic device but does not actually send the graphicalrepresentation and/or sound since the graphical representation and/orsound are stored on the server.

Consider an example in which the second user receives the sound to play,such as a recorded voice message, streaming audio, a sound clip, audiofile, or other audio from the first user. When the electronic device ofthe second user receives this sound, a DSP in this electronic deviceautomatically convolves the sound from mono or stereo sound intobinaural sound with HRTFs of the second person (e.g. customized HRTFs).When the second user hears this sound with headphones or earphones, thesound externally localizes as binaural sound outside of and away fromthe head of the second user. This SLP can be, for example, a location inempty space where no tangible object exists, a location in empty spacewhere an image exists, a location in occupied space where no electronicdevice exists (e.g., sound localizing to a stuffed animal, chair, orwall), or a location in occupied space where an electronic device exists(e.g., sound localizing to an electronic watch with no speakers).

Consider an example in which two users exchange text messages and emojisduring an electronic communication. An electronic device of the firstuser transmits a talking emoji to an electronic device of the seconduser. The second user is busy and does not immediately view the emojithat displays on the display of the second electronic device as anunread message. Immediately upon receipt of the emoji and without acommand or instruction from the second user, a digital signal processor(DSP) in the electronic device of the second user convolves or processesthe sound of the emoji from mono sound or stereo sound into binauralsound. After this convolution occurs, the second user is no longer busy,and clicks or activates the talking emoji which causes a voice of thefirst user to say “Give me a call.” This voice externally localizesabout one meter outside of and away from the head of the second user.

The sound can be obtained or generated in a variety of ways. By way ofexample, a computer or electronic device generates the sound(computer-generated sound), or microphones capture and record the soundto be sent. For example, one or more microphones capture the sound asmono sound or stereo sound when the first user speaks a message or avoice call to the second user. As another example, the first electronicdevice or a server in communication with the first electronic deviceincludes a plurality pre-recorded or previously generated sounds thatwill play to the second user.

The sound can be stored in memory of an electronic device, obtained frommemory of an electronic device (such as a computer or server), and/ortransmitted or streamed over one or more networks.

Consider an example in which the first electronic device executes amobile software messaging application that includes hundreds orthousands of sound clips or sound files in mono or stereo sound. Thefirst electronic device obtains or has access to these sound clips orsound files and can send them to other users of the mobile softwaremessaging application. The electronic device, however, may not havepermission or access to HRTFs of the other users. Hence, the electronicdevice sends the sounds to the other users in mono sound or stereosound.

Consider an example in which the first electronic device obtains thesound when the first user speaks into microphones in the firstelectronic device or in communication with the first electronic device.The microphones records the voice of the first user as he or she recordsa message or sound to be played to the second user. The first user sendsthe sound (with or without a graphical representation) to the seconduser. For example, the first user sends the actual sound file or a linkor network location to the sound. For instance, the second electronicdevice receives the network location, navigates to the location,retrieves the sound, and convolves the sound into binaural sound for thesecond user. The second electronic device convolves the sound andchanges it from mono or stereo sound into binaural sound for playing tothe second user before the second user commands or instructs theelectronic device to do so.

Consider an example in which the first and second users talk to eachother during an electronic call, telephony call, or telephone call(e.g., a Voice over Internet Protocol or VoIP call). One or moremicrophones in or in communication with the electronic device of thefirst user capture the voice of the first user. The voice transmits overthe Internet as mono sound or stereo sound to the electronic device ofthe second user. The electronic device of the second user convolves thesound into binaural sound before the second user requests to hear thesound. In fact, convolution of the sound can occur before the seconduser is even aware that he or she received sound from the first user.Alternatively, a server in communication with both electronic devicesconvolves the sound. For instance, the voices first transmit to theserver that convolves the sound and forwards the convolved sound to theelectronic device of the receiving party.

The electronic device of the second user (or another electronic device)obtains and/or retrieves the head-related transfer functions (HRTFs)used for convolution (e.g., retrieves the HRTFs of the second user). Forexample, the electronic device retrieves or receives the HRTFs of thesecond user from memory, such retrieving them from the second electronicdevice, from a server, from a database, from a network location, etc.

The HRTFs can be generic HRTFs, customized HRTFs, or HRTFs that arecustomized to the listener. Customized HRTFs or HRTFs that arecustomized to the listener are specific to an anatomy of a particularlistener and are based on a size and/or shape of the head and/or ears ofthe listener. Customized HRTFs can be obtained from actual measurements(e.g., measuring HRIRs and/or BRIRs from a head of the user) or fromcomputational modeling (e.g., modeled from a photo of the user ormodeled from measurements or approximations of the listener, such as asize and/or shape of the listener's head or ears). Customized HRTFs arealso known as individualized HRTFs.

Generic HRTFs are not specific to an anatomy of the listener. GenericHRTFs can be obtained from actual measurements (e.g., measuring HRIRsand/or BRIRs from a head of the user or a dummy head) or fromcomputation modeling. Generic HRTFs can work for a large group of peoplesince these HRTFs are not customized or individualized to each person.These HRTFs are often stored in public databases and available to thegenerally public to use free of charge.

FIG. 2 is a method that expedites playing of sound to a user byprefetching, decrypting, and/or caching the sound before the sound isplayed to the listener in accordance with an example embodiment.

In an example embodiment, blocks discussed in connection with FIG. 2 canexecute before a user, computer, or software program requests playing ofthe sound or activates the sound to play (e.g., opens or executes asound file, activates a graphical representation, etc.). For example,one or more of the blocks execute before the user hears or listens tothe sound or requests to do so. Further, the user may not know or beaware of the sound (e.g., the user receives sound to play from anotheruser via a mobile messaging application but has not logged into orchecked the application to see the received sound). Thus, the seconduser is not aware that he or she has a message waiting or sound waitingto hear.

Block 200 states receive and/or obtain sound as mono sound or stereosound.

For example, an electronic device receives or obtains the sound fromlocal memory (e.g., memory on the electronic device), local storage(e.g., memory directly attached to the electronic device), remotestorage (e.g., memory accessed over the Ethernet or wireless network), aserver, a database, a data center, etc.

For example, a first PED sends mono or stereo sound to a second PED overa wireless network (e.g., a cellular network or the Internet). Asanother example, the electronic device of the user obtains or retrievesthe sound in anticipation of the second user requesting to hear or playthe sound.

Block 210 states prefetch encrypted HRTFs from memory.

The sound can be stored and encrypted. For example, the data (e.g.,HRTFs, HRIRs, etc.) are encrypted so that only a user, computer, orsoftware program with a secret key (e.g., a decryption key) or passwordcan read the data. Encrypted data is also ciphertext, and unencrypteddata is plaintext. Encryption includes asymmetric encryption (or publickey encryption) and symmetric encryption.

The HRTFs are encrypted to protect the confidentiality of the data sounwanted third parties cannot access and/or decrypt the data. Encryptionthus protects confidentiality of a user's HRTF (e.g., customized HRTFsthat are unique to the user).

Generally, each user wants to control who or what has access to theHRTFs of the user. This enables each user to determine what entities canaccess the HRTFs of the user, especially customized HRTFs since theseare unique to each user. Users can input or provide this informationabout which entities have or do not have access to the HRTFs. Thisinformation can also be input or provided with an electronic device orsoftware program. For example, a software program automatically gathersand inputs or updates this information.

Data can be encrypted with an encryption algorithm and encryption key togenerate the ciphertext and then stored in memory. For example,symmetric cryptography uses a same key to both encrypt and decrypt thedata, while asymmetric cryptography uses two different keys (e.g., onepublic key and one private key) to encrypt the data.

Prefetching the data occurs when the computer performs fetch operationswhose result is expected to be needed soon. The prefetch occurs beforethe data is known to be needed. Examples of prefetching include cacheprefetching and prefetch input queue (PIQ).

Cache prefetching occurs when the processor increases execution byfetching instructions or data from one storage or memory location to afaster storage or memory location before the instructions or data areactually needed. For example, the data is fetched from main memory intolocal cache memory where it remains until it is needed or required. Thedata or instructions can be accessed from the cache memory faster thanthe main memory.

Cache prefetching can occur via hardware and/or software. For example,hardware prefetching occurs when the processor (or a dedicated hardwaremechanism in the processor) watches a stream of instructions or databeing requested by the executing program, recognizes the next fewelements that the program might need based on this stream, andprefetches these elements (data or instructions) into the cache memoryof the processor. Software prefetching occurs when the compiler orprocessor analyzes code and inserts an additional prefetch instructioninto the program during compilation.

PIQ includes fetching the instruction opcodes from program memory beforeor in advance of their need or request. Fetching the opcodes in advanceor prior to their need or request for execution increases the overallefficiency of the processor by boosting its execution speed. Theprocessor is not required to wait for the memory access operations forthe next instruction opcode to finish.

Block 220 states decrypt the HRTFs.

Decryption is the process of transforming data that has been encryptedback to its unencrypted form or state. Decryption is generally thereverse process of encryption. The computer or processor executes toextract and convert the encrypted or garbled data into a readable orunderstandable version. The data can be decrypted with a decryptionalgorithm based, for example, on symmetric or asymmetric cryptography.For example, data is decrypted with a secret key or password.

Block 230 states provide decrypted HRTFs to processor that convolves themono or stereo sound with the decrypted HRTFs.

Once the HRTFs are decrypted, the sound is convolved with the HRTFs totransform the sound in binaural sound for the listener.

Block 240 states move the convolved sound into local or cache memory inanticipation of the sound being played.

In anticipation of the binaural sound being requested or played, thebinaural sound is moved into local memory or cache memory. When arequest for the sound occurs, convolution is not necessary since thesound was previously convolved in anticipation of the request to play orhear the sound.

In an example embodiment, a preprocessor executes or processes the datato expedite playing, providing, or processing the binaural sound. Apreprocessor is a program that processes the retrieved data to produceoutput that is used as input to another program. This output isgenerated in anticipation of the use of the output data. For example, anexample embodiment executes instructions that predict a likelihood ofrequiring the output data and preprocesses the data in anticipation of arequest for the data. For instance, the program retrieves one or morefiles containing HRTF pairs and extracts data from the files that willbe used to convolve the sound to localize at a location corresponding tothe HRTF pair data. This extracted or preprocessed data can be quicklyprovided to a DSP in the event sound is convolved with the HRTF pair.

As another example, the processor requests a data block (or aninstruction block) from main memory before the data block is actuallyneeded. The data block is placed or stored in cache or local memory sothe data is quickly accessed and processed to externally localize soundto the user. Prefetching of this data reduces latency associated withmemory access.

Prefetching, preprocessing, decrypting, and/or caching the HRTFs canoccur or commence upon execution of an event. When the event occurs,prefetching, preprocessing, decrypting, and/or caching commences (e.g.,execute one or more blocks associated with the methods discussedherein). Examples of these events include, but are not limited to, oneor more of the following: when the user opens a software program (e.g.,a mobile messaging application or other software program that enableselectronic communication, such as telephone calls and/or messaging),when the user focuses, maximizes, or brings a window to the foreground(e.g., a mobile messaging application or other software program thatenables electronic communication), when an electronic device receives agraphical representation from another user or another electronic device(e.g., a first user sends a second user an emoji), when an electronicdevice receives sound from another user or another electronic device(e.g., a first user sends mono or stereo sound to a second user), whenan electronic device receives a voice message (e.g., an electronicdevice of the first user receives a voice message or voice mail from asecond user), when the user dons or turns on headphones or earphones(e.g., this event signifies the user may want to hear binaural sound),when the user records sound with an electronic device, when the usersends sound with an electronic device, when the user dons or turns on awearable electronic device (e.g., the user dons electronic glasses or ahead mounted display), when the user clicks or activates an icon orgraphical representation, when the user enters a virtual reality (VR)location (e.g., the user enters a VR chat room), when the user receivesor initiates a telephone call or chat or other electronic communication,when the user records a video or captures a photograph with a camera,when the user, electronic device, or software program takes anotheraction that indicates or anticipates binaural sound will be played orrequested.

Consider an example in which a mobile messaging software applicationmonitors incoming messages. When the application receives an incomingsound file or audio file (e.g., a WAV file, MP3 file, WMA file, MPEGfile, or other audio file format), the application retrieves or obtainsthe HRTFs of the user who received the audio file and convolves theaudio file into binaural sound. Thus, the act of receiving the audiofile automatically triggered, caused, or initiated the retrieval of theHRTFs, convolution of the sound, or another action (e.g., discussed inFIG. 2 or 3).

Consider an example in which a user dons a head mounted display (HMD)and enters a virtual office. A light blinking on a virtual voice messagemachine notifies the user that he or she has voice message. The voicemessage was previously received in mono sound. In anticipation of theuser activating or requesting to listen to the voice message, thesoftware application executing the virtual office prefetches the HRTFsof the user and convolves the mono sound into binaural sound that willlocalize to an image of the sender if and when the user activates thevoice message. In this example, the act of the user entering the virtualoffice or the user looking at the blinking light of the voice messagemachine initiated the actions of prefetching the HRTFs and convolvingthe sound.

FIG. 3 is a method that expedites playing of sound to a user by storingmultiple versions of the sound in memory in accordance with an exampleembodiment.

Block 300 states convolve the mono sound and/or stereo sound intobinaural sound.

A processor (such as a DSP) convolves the mono sound and/or stereo soundinto binaural sound as discussed herein.

Block 310 states store the mono sound, stereo sound, and/or binauralsound in memory for subsequent playing.

Multiple versions of the sound are simultaneously stored in memory.These versions include mono sound, stereo sound, and/or binaural sound.Storing multiple versions of the same sound expedites playing of thesound to the user. For example, if the user requests to hear the soundas binaural sound, then the sound is already convolved and/or stored andready for immediate playing. If the user requests to hear stereo sound,then the sound is already processed and/or stored and ready forimmediate playing. Likewise, if the user requests to hear mono sound,then the sound is already processed and/or stored and ready forimmediate playing.

A graphical representation can include or be associated with sound. Forexample, sound plays to the user when the user, an electronic device, ora software program activates the graphical representation or the soundassociated with the graphical representation.

Consider an example in which two users execute a mobile messagingsoftware application. The first user sends the second user an animatedemoji (or animoji) that when activated or executed says “Hello” inbinaural sound to the second user.

By way of example, sound can localize to the listener as mono sound orstereo sound when the sound is not convolved and played to the listenerwith headphones, earphones, etc. Mono sound and stereo sound can alsoexternally localize to speakers, such as speakers in a smartphone,stereo speakers in a room, etc. Alternatively, sound externallylocalizes to the listener when the sound is convolved into or capturedas binaural sound or 3D sound. Binaural sound externally localizesoutside or away from the head of the listener and is not required tolocalize to a physical object, such as a speaker. For instance, binauralsound can externally localize one or more meters away from a person at alocation in empty space (e.g., where no speaker exists or no physical ortangible object exists). Binaural sound can also localize to physicalobjects that do not have an electronic speaker, such as localizing to awall or a chair. Sound can also localize as a mix of binaural, mono, andstereo sounds, such as sound commencing as binaural sound thentransitioning to stereo sound to the listener.

If the sound is mono sound or stereo sound and not subsequentlyconvolved with HRTFs or other sound localization information (SLI), thenthe sound will not externally localize as binaural sound. For instance,a user receives a graphical representation and sound recorded in mono orstereo sound.

In an example embodiment, a sound file, sound clip, streaming sound, arecording, or other type of sound associates with or corresponds to agraphical representation. Binaural sound plays to the listener when thegraphical representation activates.

In an example embodiment, a user, a listener, a program or softwareapplication, or an electronic device activates the graphicalrepresentation and/or causes the binaural sound to play to the listener.

For example, the listener interacts with a user interface and provides acommand or instruction to play the sound upon receiving the graphicalrepresentation. For instance, the user performs one or more actions thatinclude, but are not limited to, clicking or activating an icon, emoji,graphical representation, or other indicia that represents a sound clip,sound file, streaming sound, or recording, selecting the sound from amenu (such as a dropdown menu), selecting the sound from a folder orfile (such as a folder or file being displayed to the first user),providing a body gesture (such as a hand gesture or hand movementindicating a desire to play the sound), providing head movement or eyemovement (such as the listener moving his or her head in a certaindirection or pattern to indicate selection of the sound), providing avoice command (such as the listener speaking an instruction at a naturallanguage user interface), or taking another action to have the soundplayed to the listener.

As another example, the sound automatically plays. For instance, thesound plays when the listener receives the graphical representation,opens the software program providing the graphical representation, orviews the graphical representation on a display. This sound waspreviously convolved in anticipation of the action occurring to play thesound.

As another example, the sound plays when a sender of the sound (e.g.,another user in an electronic communication with the listener) activatesthe sound or designates when the sound plays.

Binaural sound is provided to the listener through one or moreelectronic devices including, but not limited to, one or more ofheadphones, earphones, earbuds, bone conduction devices, or otherelectronic devices with speakers at, in, or near the ears of thelistener. Binaural sound can be processed for crosstalk cancellation andprovided through speakers separate or away from the listener (e.g.,dipole stereo speakers). Electronic devices in communication withheadphones, earphones, and earbuds can provide binaural sound to thelistener (e.g., a smartphone in wireless communication with earphones).

Various types of electronic devices can include or be in communicationwith speakers to provide binaural sound to listeners. Examples of theseelectronic devices include, but are not limited to, wearable electronicglasses, smartphones, head mounted displays (HMDs), optical head mounteddisplays (OHMDs), wearable electronic devices (WEDs), portableelectronic devices (PEDs), handheld portable electronic devices (HPEDs),laptop computers, tablet computers, desktop computers, and otherelectronic devices.

From the point-of-view of the listener, the sound originates or emanatesfrom an object, point, area, or direction. This location for the originof the sound is the sound localization point (SLP). By way of example,the SLP can be an actual point in space (e.g., an empty point in space1-2 meters away from the head of the listener) or a point on or at aphysical or virtual object (e.g., a mouth or head of an augmentedreality (AR) or virtual reality (VR) image). The SLP does not have to beso precise since humans are not always able to localize sound to aparticle point. As such, the SLP can also be a specific or general area(e.g., a location next to and on the right side of the listener) or aspecific or general direction from where the sound originates to thelistener (e.g., a location several meters behind the listener).

When binaural sound is provided to the listener, the listener will hearthe sound as if it originates from the sound source, the source ofsound, or the SLP. The sound, however, does not originate from the soundsource since the sound source or SLP may be an inanimate object with noelectronics or an animate object with no electronics. Alternatively, thesound source or SLP has electronics but does not have the capability togenerate sound (e.g., the sound source has no speakers or sound system).As yet another example, the sound source or SLP has speakers and theability to provide sound but is not providing sound to the listener. Ineach of these examples, the listener perceives the sound to originatefrom the sound source or SLP, but the sound source or SLP does notproduce the sound. Instead, the sound is altered or convolved andprovided to the listener so the sound appears to originate from thesound source or SLP.

In an example embodiment, at least a portion of the sound associatedwith, corresponding to, or provided from the graphical representationexternally localizes away from the head of the listener in empty space(e.g., where no physical or tangible object exists) or occupied space.For example, the sound externally localizes proximate or near thelistener, such as localizing within a few meters of the listener. Forinstance, the SLP where the listener localizes the sound is stationaryor fixed in space (e.g., fixed in space with respect to the user, fixedin space with respect to an object in a room, fixed in space withrespect to an electronic device, fixed in space with respect to anotherobject or person).

By way of example, the SLP can be an actual point in space (e.g., anempty point in space 1-2 meters away from the head of the listener) or apoint on a physical or virtual object (e.g., a mouth or head of anaugmented reality (AR) or virtual reality (VR) image). The SLP does nothave to be so precise since humans are not always able to localize soundto a particle point. As such, the SLP can also be a general area (e.g.,a location next to and on the right side of the listener) or a generaldirection from where the sound originates to the listener (e.g., alocation several meters behind the listener).

Consider an example in which the graphical representation is an emojithat includes a talking animated animal head or human head. When alistener clicks on or activates the emoji, the head talks and thelistener hears the voice as binaural sound that externally localizesabout one meter away from the listener. For instance, the voice isconvolved with head-related transfer functions (HRTFs) having sphericalcoordinates (distance r=1.0 m, elevation ϕ=0°, azimuth θ=) 30°. Thelistener activates the emoji and hears the voice originate fromspherical coordinates (1.0, 0°, 30°).

By way of example, a computer or electronic device generates the sound(computer-generated sound), or microphones capture and record the soundto be sent. For example, one or more microphones capture the sound asmono sound or stereo sound when the first user speaks a message to thesecond user. As another example, the first electronic device or a serverin communication with the first electronic device includes a pluralitypre-recorded or previously generated sounds.

Consider an example in which the first electronic device executes amobile software messaging application that includes hundreds orthousands of sound clips or sound files. The first electronic deviceobtains or has access to these sound clips or sound files and can sendthem to other users of the mobile software messaging application.

Consider an example in which the first electronic device obtains thesound when the first user speaks into microphones in the firstelectronic device or in communication with the first electronic device.The microphones records the voice of the first user as he or she recordsa message or sound to be played to the second user.

The first electronic device transmits the sound and a graphicalrepresentation associated with or corresponding to the sound over one ormore wired or wireless networks (e.g., a cellular network, the internet,etc.). For example, the first electronic device includes a wirelesstransmitter/receiver that sends the sound and graphical representation.

Consider an example in which the first user commands or instructs thesound clip to play to the second user during an electronic communicationbetween the first and second users. In response to this command orinstruction, the first electronic device transmits the sound clip and a3D moving emoji to the second electronic device.

In another example embodiment, a server or another electronic devicetransmits the sound and/or graphical representation to the secondelectronic device. Consider an example in which the first and secondusers talk or message each other with a mobile messaging softwareapplication. The application executes on the electronic devices and oneor more servers. When the first user clicks on a 3D sound emoji, thisaction causes one of the servers to transmit the 3D emoji and sound tothe second electronic device which receives and convolves the soundbefore the second user requests to hear the sound.

The second electronic device receives the sound and the graphicalrepresentation from the first electronic device or another electronicdevice (e.g., a server) in communication with the first electronic. Forexample, the second electronic device includes a wirelesstransmitter/receiver that receives the sound and graphicalrepresentation over one or more networks.

A processor or sound hardware processes or convolves the sound withhead-related transfer functions (HRTFs) or other SLI so the sound willexternally localize as binaural sound to the listener.

Graphical representations can have many sizes, shapes, and forms (e.g.,people, faces, characters, animals, objects, 2D, 3D, etc.). Further, thegraphical representations can be static, such as a 2D or 3D emoji thatdo not move or change facial expressions. Alternatively, the graphicalrepresentations can be dynamic, such as 2D or 3D emoji that move, talk,change facial expressions, rotate, etc. Further yet, graphicalrepresentations in accordance with example embodiments can be presentedas AR images and VR images.

The graphical representations can include or be associated with sound,such as a sound clip, a sound file, a recorded voice message, streamingaudio, etc. The sound can play for a short period of time (e.g., lessthan one second, one second, two seconds, etc.). For example, the soundis a voice saying “Hello” or “Hahahaha” or “Thank you” or another shortaudio message. As another example, the sound is a computer-generated“Beep” or phone ringing or explosion sound. The sound can play forlonger periods of time (e.g., ten seconds, thirty seconds, one minute,several minutes, etc.). For example, the sound is a recorded messagefrom a user during an electronic communication between two users.

By way of example, the sound plays when the listener activates thegraphical representation or another action occurs that initiates oractivates playing of the sound. For example, a first user sends a seconduser an emoji shaped like a heart. This heart appears on a display of anelectronic device of the second user. When the second user clicks on theheart, a voice in binaural sound says “I love you” to the second user.

When sound is already convolved into binaural sound, this sound can beconverted back into mono or stereo sound or played as mono or stereosound. For example, the electronic device plays the sound through asingle speaker. As another example, the electronic device plays the samechannel through both speakers (e.g., play the left channel sound to boththe left and right speakers of the headphones or play the right channelsound to both the left and right speakers of the headphones). As anotherexample, the sound is filtered through cross-talk canceling filters.Filters, for example, can eliminate crosstalk and the HRTFs (e.g., byutilizing an inverse filter, such as a Nelson/Kirkeby inverse filter).

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Theelectronic device of the second user consults privacy settings todetermine whether the first user is authorized to have access tocustomized HRTFs of the second user. In response to consulting theseprivacy settings, the electronic device of the second user denies orallows access to the customized HRTFs. For example, the electronicdevice denies access to the customized HRTFs of the second user when theprivacy settings indicate that the first user is not authorized to haveaccess to the customized HRTFs of the second user. For example, theelectronic device allows access to the customized HRTFs of the seconduser when the privacy settings indicate that the first user isauthorized to have access to the customized HRTFs of the second user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user has authorization toprovide binaural sound to the second user. If the first user has thisauthorization, then mono or stereo sound received from the first user isconvolved into binaural sound and played to the second user. If thefirst user does not have this authorization, then the sound plays to thesecond user as mono sound or stereo sound.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user is authorized toprovide sound to the second user in the binaural sound. The voice of thefirst user plays to the second user in the binaural sound when the firstuser is authorized to provide the sound to the second user in thebinaural sound. The voice of the first user is changed from being in thebinaural sound to being in one of mono sound and stereo sound when thefirst user is not authorized to provide the sound to the second user inthe binaural sound.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user is authorized to sendhis or her voice to the second user in the binaural sound. The voice ofthe first user changes from being provided to the second user inbinaural sound to being provided to the second user in one of mono soundand stereo sound upon determining that the first user is not authorizedto send the voice to the second user in the binaural sound.

In an example embodiment, the electronic device display the graphicalrepresentation with one or more of the following: an indication that thesound will externally localize as binaural sound to the second user, anindication of a location where the sound will externally localize asbinaural sound to the second user, and an indication informing thesecond user to wear headphones or earphones before listening to thesound.

FIGS. 4A-4F show a plurality of graphical representations with one ormore indications in accordance with example embodiments. By way ofexample, the graphical representations 400A-400F are shown as faces,such as a face of an emoji, emoticon, etc. Such faces can have manyshapes and forms, such as human faces, cartoon character faces, animalfaces, animated faces, etc.

Example embodiments are not limited to graphical representations thatinclude faces, such as those shown in FIGS. 4A-4F. Graphicalrepresentations can have many sizes, shapes, and forms (e.g., people,faces, characters, animals, and objects).

Furthermore, these graphical representations are shown astwo-dimensional but can also be three-dimensional (3D). Further, thegraphical representations can be static, such as a 2D or 3D emoji thatdo not move or change facial expressions. Alternatively, the graphicalrepresentations can be dynamic, such as 2D or 3D emoji that move, talk,change facial expressions, rotate, etc. Further yet, graphicalrepresentations in accordance with example embodiments can be presentedas AR images and VR images.

The graphical representations 400A-400F include or are associated with asound, such as a sound clip, a sound file, a recorded voice message,streaming audio, etc. The sound can play for a short period of time(e.g., less than one second, one second, two seconds, etc.). Forexample, the sound is a voice saying “Hello” or “Hahahaha” or “Thankyou” or another short audio message. As another example, the sound is acomputer-generated “Beep” or phone ringing or explosion sound. The soundcan play for longer periods of time (e.g., ten seconds, thirty seconds,one minute, several minutes, etc.). For example, the sound is a recordedmessage from a user during an electronic communication between twousers.

By way of example, the sound plays when the listener activates thegraphical representation or another action occurs that initiates oractivates playing of the sound. For example, a first user sends a seconduser an emoji shaped like a heart. This heart appears on a display of anelectronic device of the second user. When the second user clicks on theheart, a voice in binaural sound says “I love you” to the second user.

With example embodiments, the graphical representations can represent orsymbolize the listener or source of sound (depending on whatinstructions or understandings are provided to the listener and/orusers). The graphical representations and indications are displayed tothe listener on a display of a wearable electronic device (WED),portable electronic device (PED), handheld portable electronic device(HPED), head mounted display (HMD), or other electronic device discussedherein. The electronic device and display are not shown in FIGS. 4A-4Ffor ease of illustration. Further, these figures are shown from thepoint-of-view of the listener looking at the display and/or interactingwith the electronic device.

FIG. 4A shows a graphical representation 400A with a visual indication410A of a location of where binaural sound localizes. The indicationincludes three circles that represent SLPs near the head or face of thegraphical representation which is understood to represent the head ofthe second user. Each circle represents a different SLP where the usercan select to have binaural sound externally localize away from the headof the second user. One SLP 412A appears in front of and on a right sideof the head of the graphical representation (representing the head ofthe second user); one SLP 414A appears in front of and above the head ofthe graphical representation (representing the head of the second user);one SLP 416A appears in front of and on a left side of the head of thegraphical representation (representing the head of the second user).

Consider an example in which the graphical representation 400A displayson or through an electronic device of the first user (e.g., a HMD,smartphone, or wearable electronic device). The first user selects oneof the indications 412A, 414A, or 416A and transmits the graphicalrepresentation 400A and mono sound to the second user during anelectronic communication between the first user and the second user.When the electronic device of the second user receives the graphicalrepresentation, this electronic device extracts the coordinates of theselected SLP, selects the corresponding coordinates of HRTFs, convolvesthe sound with the selected HRTFs, and plays the sound to the seconduser as binaural sound to the location of the indication selected by thefirst user. For example, if the first user selected 412A, then thebinaural sound originates in front of and to a right side of the head ofthe second user since the location of 412A displayed to the first userwas in front of and to a right side of the head of the graphicalrepresentation 400A.

Alternatively, the second user selects one of the SLPs 412A, 414A, or416A as the location where the binaural sound will localize to thesecond user. Further, the selected SLP can be highlighted (e.g., withcolor or light) to provide a visual indication on the display as towhere the binaural sound will externally localize or is externallylocalizing to the second user.

FIG. 4B shows a graphical representation 400B with a visual indication410B displayed on or with the display of the second user. The indicationrepresents a SLP or location where binaural sound will emanate or isemanating with respect to the listener. The indication 410B is locatedin front of the face or head and shows a relative location where thebinaural sound will originate. For example, the binaural sound willlocalize to a SLP that is in empty space about one meter away from thehead or face of the listener as shown in FIG. 4B.

FIG. 4C shows a graphical representation 400C with a visual indication410C. The indication includes three symbols (“3D”) that representthree-dimensional or binaural sound. Each 3D symbol represents SLPs nearthe head or face of the graphical representation which is understood torepresent the head of the second user. Each 3D symbol represents adifferent SLP where the first or second user can select to have binauralsound externally localize away from the head of the second user. One 3Dsymbol 412C appears in front of and slightly above the face of thegraphical representation (representing the head of the second user); one3D symbol 414C appears directly front of the face of the graphicalrepresentation (representing the head of the second user); one 3D symbol416C appears in front of and slightly below the face of the graphicalrepresentation (representing the head of the second user).

Consider an example in which the graphical representation 400C displayson or through an electronic device of the second user (e.g., a HMD,smartphone, or wearable electronic device). The second user sees threedifferent locations for where he or she can select to have binauralsound localize. The 3D symbols visually inform the second user that thesound is binaural sound. The second user selects one of the indications412C, 414C, or 416C, and sound plays to the location selected.

An example embodiment convolves the sound to each of the three SLPlocations before the second user makes the selection. Since there areonly a limited number of SLP locations available, the computer programknows that the second user will select one of these locations (assumingthe second user desires to hear the sound as binaural sound). When thesecond user makes a selection of the one of the SLPs, the sound isalready convolved and ready for immediate play to the second user. Forexample, if the second user selects 414C, then the binaural soundoriginates directly in front of face of the second user.

FIG. 4D shows a graphical representation 400D with an indication 410D.The indication includes a menu that enables the user to select how soundwill play to the listener. The menu options includes playing the soundas binaural sound, mono sound, or stereo sound. The option “stereo”sound is bolded to indicate this is the option selected by the user.Here, the user has three choices or options for how to hear the sound.

Consider an example in which the first user sends the graphicalrepresentation 400D to the second user. The first user selects theoption “binaural sound” from the menu and transmits the graphicalrepresentation to the second user. When the electronic device plays thesound of the graphical representation to the second user, the seconduser hears the sound as binaural sound since this was the selection ofthe first user. The sound transmits with mono sound, but the selectionof the first user causes the DSP in the electronic device of the seconduser to convolve the sound upon receipt before the second user requestsor acts to hear the sound.

Consider an example in which the first user sends the graphicalrepresentation 400D to the second user, but the first user makes noselection (e.g., the sound is sent as mono sound with no indication howthe second user will hear the sound). The graphical represent displaysto the second user who can select how to hear the sound. Alternatively,a computer program makes the selection and notifies the user. Forexample, the selection “stereo” is bolded to indicate this is how thesound will localize to the user.

FIG. 4E shows a graphical representation 400E with an indication 410E.The indication includes two options for how or where the sound willlocalize. One option (“3D”) visually instructs the user that the soundwill localize as 3D sound or binaural sound. Another option (“Mono”)visually instructs the user that the sound will localize as mono sound.

Consider an example in which the first user records a voice message asmono sound and sends this voice message as the graphical representation400E to the second user. The graphical representation is a moving 3Dimage of the face and head of the first user. The second user has anoption to hear the sound as mono sound or binaural sound. Before makingthis selection, the computer program does not know which selection thesecond user will make. In anticipation of the second user selectingbinaural sound (i.e., selecting the 3D symbol), the computer programconvolves the mono sound into binaural sound. The computer program thensaves both the sound as mono sound and binaural sound. If the seconduser subsequently selects mono sound, then the sound is ready to play tothe second user. On the other hand, if the second user selects 3D sound,then the binaural sound is ready for immediate play to the second userwithout having to convolve the sound since the convolution alreadyoccurred.

FIG. 4F shows a graphical representation 400F with an indication 410F.The indication includes a sound wave (shown with three squiggly orwaving lines) and the phrase “3D Sound.” The indication shows the user adirection and location for the origination of the binaural sound. Theuser is thus able to see (in advance of hearing the sound) where thesound will originate.

The indications can thus serve as a way to visually inform users thatthe sound associated with the graphical representation will be binauralsound. Users learn the recognize the indication as a symbol for binauralsound. When a listener sees the indication, he or she immediately knowsin advance that the sound will be binaural sound and externallylocalize, as opposed to mono sound or stereo sound that internallylocalizes inside a head of the listener.

Consider an example in which a first user and a second user talk orexchange talking graphical representations during an electroniccommunication. The first user sends the second user a talking emoji thatdisplays to the second user on a HPED. This emoji is an animated headthat looks like or represents the first user. So, the face of the firstuser appears on the display of the HPED of the second user and faces thesecond user. The indication also appears on this display and shows thesecond user that the sound will be binaural sound.

The graphical representation and/or indication provides the listenerwith a variety of different valuable information. Consider the examplein which the graphical representation represents or symbolizes the headof the listener and is shown on a display of an HPED to the listener.First, the indication shows the listener that the sound will be binauralsound since the location of the indication is physically located outsideof the head of the graphical representation. Second, the indicationshows a location of where this binaural sound will initially localize tothe listener because the indication is positioned relative to thegraphical representation at the same relative location with respect tothe second user.

The location of the source of binaural sound can appear inside the bodyof the graphical representation with words, text, symbols, images, orother indicia that indicate a direction and/or distance to the source ofthe sound. For example, the indication includes a compass heading (suchas North, South, East, or West) or a coordinate location (such ascoordinate location in rectangular coordinates, polar coordinates, orspherical coordinates).

In an example embodiment, the indication remains displayed with thegraphical representation. For instance, while the graphicalrepresentation displays to the listener, the indication simultaneouslydisplays to the listener. In another example embodiment, the indicationdisplays for a temporary period of time with the graphicalrepresentation. For instance, the indication initially displays with thegraphical representation to notify or inform the user of the existenceand location of the source of the binaural sound. The indication thendisappears while the graphical representation continues to display tothe listener while the sound plays to the listener.

The indication includes the acronym or letters “3D” that stand forthree-dimensional. Based on this indication, the listener expects thesound to be 3D sound or binaural sound that externally localizes to thelistener.

Consider an example embodiment in which the indication is instead “Mono”or “Stereo” or another symbol or word to visually indicate that thesound will localize as or be provided as mono sound or stereo sound.

FIG. 5 is an example computer system 500 in accordance with an exampleembodiment.

The computer system 500 includes one or more of a server 510, a database520, a database 524, an electronic device 530, and an electronic device540 in communication over one or more networks 550. User 539 is with oruses electronic device 530, and user 549 is with or uses electronicdevice 540. For illustration, a single server 510, two databases 520 and524, two electronic devices 530 and 540, and two users 539 and 549 areshown, but example embodiments can include a plurality of servers,databases, electronic devices, and users.

Server 510 includes a memory 512 and a processing unit 514. The server510 couples to or communicates with the database 520 that includes HRTFsand other sound localization information 522 and database 524 thatincludes graphical representations (reps) 526 and sound clips 528.

Electronic device 530 includes a processing unit 532 and memory 534 withHRTFs 536 and cache 538.

Electronic device 540 includes a processing unit 542 and memory 544 withHRTFs 546 and cache 548.

FIG. 6 is an example of an electronic device 600 in accordance with anexample embodiment.

The electronic device 600 includes a processor or processing unit 610,memory 620 with sound clips 622, graphical representations or graphicalreps 624, and cache 626, a display 630, one or more interfaces 640, awireless transmitter/receiver 750, speakers 660, one or more microphones670, head tracking 680 (such as one or more of an inertial sensor,accelerometer, gyroscope, and magnetometer), and HRTFs 690 (which arestored in memory), a prefetcher 692 (that executes prefetching asdiscussed herein), encryption/decryption 694 (that encrypts and decryptsdata as discussed herein), and a mobile messaging application 696.

Mobile messaging applications are applications and/or platforms thatenable one or more messaging/chatting, talking, sending/receivinggraphical representations, file sharing, and various other forms ofelectronic communication. Such application can execute on HPEDs, PED,HMDs, and other electronic devices.

Memory includes computer readable medium (CRM).

Examples of an interface include, but are not limited to, a networkinterface, a graphical user interface, a natural language userinterface, a natural user interface, a phone control interface, areality user interface, a kinetic user interface, a touchless userinterface, an augmented reality user interface, and/or an interface thatcombines reality and virtuality.

Sound clips include sound files, sounds, recorded messages (such asvoice messages or other recorded sound), computer-generated sounds, andother sound discussed herein. For example, users can record, exchange,and/or transmit sound clips or sounds. These sound include sendingstreaming sounds or sounds in real-time during an electroniccommunication.

The processor or processing unit includes a processor and/or a digitalsignal processor (DSP). For example, the processing unit includes one ormore of a central processing unit, CPU, digital signal processor (DSP),microprocessor, microcontrollers, field programmable gate arrays (FPGA),application-specific integrated circuits (ASIC), etc. for controllingthe overall operation of memory (such as random access memory (RAM) fortemporary data storage, read only memory (ROM) for permanent datastorage, and firmware).

Consider an example embodiment in which the processing unit includesboth a processor and DSP that communicate with each other and memory andperform operations and tasks that implement one or more blocks of theflow diagram discussed herein. The memory, for example, storesapplications, data, programs, sound clips, algorithms (includingsoftware to implement or assist in implementing example embodiments) andother data.

For example, a processor or DSP executes a convolving process with theretrieved HRTFs or HRIRs (or other transfer functions or impulseresponses) to process sound clips so that the sound is adjusted, placed,or localized for a listener away from but proximate to the head of thelistener. For example, the DSP converts mono or stereo sound to binauralsound so this binaural sound externally localizes to the user. The DSPcan also receive binaural sound and move its localization point, add orremove impulse responses (such as RIRs), and perform other functions.

For example, an electronic device or software program convolves and/orprocesses the sound captured at the microphones of an electronic deviceand provides this convolved sound to the listener so the listener canlocalize the sound and hear it. The listener can experience a resultinglocalization externally (such as at a sound localization point (SLP)associated with near field HRTFs and far field HRTFs) or internally(such as monaural sound or stereo sound).

The memory stores HRTFs, HRIRs, BRTFs, BRIRs, RTFs, RIRs, or othertransfer functions and/or impulse responses for processing and/orconvolving sound. The memory can also store instructions for executingone or more example embodiments. Further, the memory can store thesound, graphical representations, and other information and instructionsdiscussed herein.

The electronic device provides sound to the users through one or morespeakers. Alternatively or in addition to the speakers, the electronicdevice can communicate with headphones, earphones, earbuds, boneconduction devices, or another electronic device that provides sound tothe user.

The networks include one or more of a cellular network, a public switchtelephone network, the Internet, a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), a personal areanetwork (PAN), home area network (HAM), and other public and/or privatenetworks. Additionally, the electronic devices need not communicate witheach other through a network. As one example, electronic devices coupletogether via one or more wires, such as a direct wired-connection. Asanother example, electronic devices communicate directly through awireless protocol, such as Bluetooth, near field communication (NFC), orother wireless communication protocol.

By way of example, a computer and an electronic device include, but arenot limited to, handheld portable electronic devices (HPEDs), wearableelectronic glasses, electronic or smart watches, wearable electronicdevices (WEDs), smart earphones or hearables, electronic devices withcellular or mobile phone capabilities or subscriber identificationmodule (SIM) cards, desktop computers, servers, portable computers (suchas tablet and notebook computers), smartphones, head mounted displays(HMDs), optical head mounted displays (OHMDs), headphones, and otherelectronic devices with a processor or processing unit, a memory, a DSP.

Example embodiments are not limited to HRTFs but also include othersound transfer functions and sound impulse responses including, but notlimited to, head related impulse responses (HRIRs), room transferfunctions (RTFs), room impulse responses (RIRs), binaural room impulseresponses (BRIRs), binaural room transfer functions (BRTFs), headphonetransfer functions (HPTFs), etc.

Example embodiments can be executed with one or more integrated circuitsthat are specifically customized, designed, or configured to execute oneor more blocks discussed herein. For example, the electronic devicesinclude a specialized or custom processor or microprocessor orsemiconductor intellectual property (SIP) core or digital signalprocessor (DSP) with a hardware architecture optimized for convolvingsound and executing one or more example embodiments.

Consider an example in which the HPED (including headphones) includes acustomized or dedicated DSP that executes one or more blocks discussedherein (including processing and/or convolving sound into binaural soundfor sound clips). Such a DSP has a better power performance or powerefficiency compared to a general-purpose microprocessor and is moresuitable for a HPED or WED due to power consumption constraints of theHPED or WED. The DSP can also include a specialized hardwarearchitecture, such as a special or specialized memory architecture tosimultaneously fetch or pre-fetch multiple data and/or instructionsconcurrently to increase execution speed and sound processing efficiencyand to quickly correct errors while sound externally localizes to theuser. By way of example, streaming sound data (such as sound data in atelephone call or software game application) is processed and convolvedwith a specialized memory architecture (such as the Harvard architectureor the Modified von Neumann architecture). The DSP can also provide alower-cost solution compared to a general-purpose microprocessor thatexecutes digital signal processing and convolving algorithms. The DSPcan also provide functions as an application processor ormicrocontroller. The DSP can also prefetch sound clips and other soundfrom memory to expedite convolution.

Consider an example in which a customized DSP includes one or morespecial instruction sets for multiply-accumulate operations (MACoperations), such as convolving with transfer functions and/or impulseresponses (such as HRTFs, HRIRs, BRIRs, et al.), executing Fast FourierTransforms (FFTs), executing finite impulse response (FIR) filtering,and executing instructions to increase parallelism.

Consider another example in which sound clips, graphicalrepresentations, and/or HRTFs (or other transfer functions or impulseresponses) are stored or cached in the DSP memory or local memoryrelatively close to the DSP to expedite binaural sound processing.

As used herein, “customized HRTFs” or “HRTFs that are customized” arespecific to an anatomy of a particular listener and are based on a sizeand/or shape of the head and/or ears of the listener.

As used herein, an “emoji” is a graphical representation that includesimages, symbols, or icons sent between users in electroniccommunications (such as text messages, e-mail, and social media) toexpress an emotional attitude of the writer, convey information, orcommunicate an message. Emojis can provide sound when activated orexecuted.

A “talking emoji” is an emoji that talks (e.g., with one or more words).

As used herein, “empty space” is a location that is not occupied by atangible object.

As used herein, “graphical representations” include, but are not limitedto, emoji, emoticons, animoji, icons, stickers, folders, documents,files, text or words, pictures, images, and other visible indicia thatdisplay on, thru, or with an electronic device. Furthermore, thesegraphical representations can be two-dimensional (2D), three-dimensional(3D), virtual reality (VR) images, augmented reality (AR) images, staticor non-moving, moving, and other types of images.

As used herein, “headphones” or “earphones” include a left and rightover-ear ear cup, on-ear pad, or in-ear monitor (IEM) with one or morespeakers or drivers for a left and a right ear of a wearer. The left andright cup, pad, or IEM may be connected with a band, connector, wire, orhousing, or one or both cups, pads, or IEMs may operate wirelessly beingunconnected to the other. The drivers may rest on, in, or around theears of the wearer, or mounted near the ears without touching the ears.

As used herein, the word “proximate” means near. For example, binauralsound that externally localizes away from but proximate to a userlocalizes within three meters of the head of the user.

As used herein, a “sound localization point” or “SLP” is a locationwhere a listener localizes sound. A SLP can be internal (such asmonaural sound that localizes inside a head of a listener), or a SLP canbe external (such as binaural sound that externally localizes to a pointor an area that is away from but proximate to the person or away frombut not near the person). A SLP can be a single point such as onedefined by a single pair of HRTFs or a SLP can be a zone or shape orvolume or general area. Further, in some instances, multiple impulseresponses or transfer functions can be processed to convolve sounds to aplace within the boundary of the SLP. In some instances, a SLP may nothave access to a particular HRTF necessary to localize sound at the SLPfor a particular user, or a particular HRTF may not have been created. ASLP may not require a HRTF in order to localize sound for a user, suchas an internalized SLP, or a SLP may be rendered by adjusting an ITDand/or ILD or other human audial cues.

As used herein, “sound localization information” or “SLI” is informationthat is used to process or convolve sound so the sound externallylocalizes as binaural sound to a listener.

As used herein, a “telephone call,” or a “electronic call” is aconnection over a wired and/or wireless network between a calling personor user and a called person or user. Telephone calls can use landlines,mobile phones, satellite phones, HPEDs, voice personal assistants(VPAs), computers, and other portable and non-portable electronicdevices. Further, telephone calls can be placed through one or more of apublic switched telephone network, the internet, and various types ofnetworks (such as Wide Area Networks or WANs, Local Area Networks orLANs, Personal Area Networks or PANs, Campus Area Networks or CANs,etc.). Telephone calls include other types of telephony including Voiceover Internet Protocol (VoIP) calls, internet telephone calls, in-gamecalls, telepresence, etc.

As used herein, a “user” or a “listener” is a person (i.e., a humanbeing). These terms can also be a software program (including an IPA orIUA), hardware (such as a processor or processing unit), an electronicdevice or a computer (such as a speaking robot or avatar shaped like ahuman with microphones in its ears or about six inches apart).

In some example embodiments, the methods illustrated herein and data andinstructions associated therewith, are stored in respective storagedevices that are implemented as computer-readable and/ormachine-readable storage media, physical or tangible media, and/ornon-transitory storage media. These storage media include differentforms of memory including semiconductor memory devices such as DRAM, orSRAM, Erasable and Programmable Read-Only Memories (EPROMs),Electrically Erasable and Programmable Read-Only Memories (EEPROMs) andflash memories; magnetic disks such as fixed and removable disks; othermagnetic media including tape; optical media such as Compact Disks (CDs)or Digital Versatile Disks (DVDs). Note that the instructions of thesoftware discussed above can be provided on computer-readable ormachine-readable storage medium, or alternatively, can be provided onmultiple computer-readable or machine-readable storage media distributedin a large system having possibly plural nodes. Such computer-readableor machine-readable medium or media is (are) considered to be part of anarticle (or article of manufacture). An article or article ofmanufacture can refer to a manufactured single component or multiplecomponents.

Blocks and/or methods discussed herein can be executed and/or made by auser, a user agent (including machine learning agents and intelligentuser agents), a software application, an electronic device, a computer,firmware, hardware, a process, a computer system, and/or an intelligentpersonal assistant. Furthermore, blocks and/or methods discussed hereincan be executed automatically with or without instruction from a user.

What is claimed is:
 1. A portable electronic device (PED) of a user, thePED comprising: a receiver that receives a talking emoji from a secondPED of a second user; and a processor that expedites playing sound ofthe talking emoji to the user by processing the sound of the talkingemoji with head-related transfer functions (HRTFs) to change the soundfrom mono sound or stereo sound into binaural sound before the useractivates the talking emoji to hear the binaural sound.
 2. The PED ofclaim 1 further comprising: a memory that stores the HRTFs as encrypted,wherein the PED retrieves the HRTFs that are encrypted before receivinga request from the user to activate the talking emoji to hear thebinaural sound, decrypts the HRTFs, and provides the HRTFs after beingdecrypted to a digital signal processor (DSP).
 3. The PED of claim 1further comprising: a display that displays the talking emoji with asymbol that visually indicates that the binaural sound of the talkingemoji will play to the user as the binaural sound and not as the monosound or the stereo sound.
 4. The PED of claim 1 further comprising: auser interface that receives a click to activate playing of the talkingemoji in the binaural sound, wherein the binaural sound of the talkingemoji plays without further convolution after receiving the click sincethe processor already processed the mono sound or the stereo sound intothe binaural sound before receiving the click to activate playing of thetalking emoji.
 5. The PED of claim 1 further comprising: a cache memorythat stores the binaural sound, wherein the PED moves the binaural soundinto the cache memory before receiving a request from the user toactivate the talking emoji and before playing the binaural sound of thetalking emoji to the user.
 6. The PED of claim 1 further comprising: amemory that stores the HRTFs, wherein the PED expedites playing thebinaural sound by retrieving the HRTFs from the memory and processingthe mono sound or the stereo sound into the binaural sound in responseto the user opening a mobile messaging application.
 7. The PED of claim1 further comprising: a memory that stores the talking emoji in the monosound or the stereo sound and simultaneously stores the talking emoji inthe binaural sound so the sound of the talking emoji is playable to thesecond person as the binaural sound and the mono sound or the stereosound.
 8. A method comprising: receiving, at a portable electronicdevice (PED) of a user, a talking emoji; and expediting playing of soundof the talking emoji to the user by processing the sound of the talkingemoji with head-related transfer functions (HRTFs) to change the soundfrom mono sound or stereo sound into binaural sound before the useractivates the talking emoji to hear the binaural sound.
 9. The method ofclaim 8 further comprising: storing, in memory of the PED, the HRTFs asencrypted; retrieving, from the memory, the HRTFs that are encryptedbefore receiving a request from the user to play the talking emoji;decrypting the HRTFs before receiving the request from the user to playthe talking emoji; and providing the HRTFs that are decrypted to theprocessor before receiving the request from the user to play the talkingemoji.
 10. The method of claim 8 further comprising: displaying, with adisplay of the PED, the talking emoji with a symbol that visuallyindicates that the sound of the talking emoji will play to the user asthe binaural sound.
 11. The method of claim 8 further comprising:receiving, at the PED, a click at the talking emoji as a request fromthe user to play the talking emoji, wherein the PED expedites playing ofthe sound of the talking emoji plays as the binaural sound uponreceiving the click since the processor already processed the sound fromthe mono sound or the stereo sound into the binaural sound.
 12. Themethod of claim 8 further comprising: expediting playing of the sound ofthe talking emoji by moving the binaural sound into cache memory in thePED before receiving a request from the user to play the talking emoji.13. The method of claim 8 further comprising: expediting playing of thesound of the talking emoji by moving the binaural sound of the talkingemoji into cache memory of the PED in anticipation of the usersubsequently activating the talking emoji to hear the sound of thetalking emoji.
 14. The method of claim 8 further comprising: storing, inmemory of the PED, the talking emoji in the stereo sound and the talkingemoji in the binaural sound so the sound of the talking emoji isplayable to the user as both the stereo sound and the binaural sound.15. A portable electronic device (PED) of a user, the PED comprising: areceiver that receives a talking emoji; and a processor that expeditesplaying sound of the talking emoji to the user by convolving the soundof the talking emoji with head-related transfer functions (HRTFs) tochange the sound from mono sound or stereo sound into binaural soundbefore the user activates the talking emoji to play the sound of thetalking emoji.
 16. The PED of claim 15 further comprising: a memory thatstores the sound of the talking emoji in the PED as the binaural soundand the mono sound or the stereo sound so the user has an option to hearthe sound of the talking emoji as the binaural sound and as the monosound or the stereo sound.
 17. The PED of claim 15 further comprising: acache memory, wherein the PED expedites playing of the sound of thetalking emoji by moving the binaural sound of the talking emoji into thecache memory in anticipation of the user subsequently activating thetalking emoji to hear the sound of the talking emoji.
 18. The PED ofclaim 15 further comprising: a memory that stores the HRTFs, wherein thePED expedites playing the sound of the talking emoji by prefetching theHRTFs from the memory in response to opening of a mobile messagingapplication that receives and plays talking emojis to the user.
 19. ThePED of claim 15, wherein the talking emoji is a three-dimensional (3D)moving emoji that is an augmented reality (AR) or virtual reality (VR)image, and the sound of the talking emoji is a recorded voice of amessage to the user from another user.
 20. The PED of claim 15 furthercomprising: a memory that stores the HRTFs as encrypted, wherein the PEDexpedites playing of the sound of the talking emoji by retrieving theHRTFs from the memory and decrypting the HRTFs in response to opening ofa mobile messaging application that receives and plays talking emojis tothe user.